Testing and Performance Evaluation of Fixed Terrestrial 3D Laser Scanning Systems for Highway Applications

In many 3D laser scanner applications, high relative precision (relative dimensions within the registered point cloud) is sufficient; in contrast, DOT applications require good relative precision and high absolute accuracy. Thus, precise and robust geo-referencing is critical, and robust workflows are needed to reduce the likelihood and impact of human errors. DOT applications have a fairly unique combination of challenges, including longer range requirements, long linear geometry for complete jobs, tall structures, and dark pavement scanned at high incidence angle. These factors motivate standardized protocols and metrics for characterizing and evaluating scanner performance and to develop confidence limits for the scanner data in DOT applications. The paper’s primary contribution is a set of vendor-neutral standard test protocols for the characterization and evaluation of 3D laser scanner performance, which users can conduct in easily accessible facilities. These evaluations focused on issues significant in DOT survey applications, workflows, and data flows. Example performance evaluation is provided for several commercially available 3D laser scanners. This paper provides the needed scientific basis for data-driven deployment of this valuable measurement tool. The paper also provides recommendations and guidelines which will promote consistent and correct use of 3D laser scanners by DOTs and their contractors. The guidelines clarify the common limitations of 3D laser scanners and recommend mitigation methods; this will help engineers and surveyors to select the right scanner and determine optimum scanning settings for survey applications. INTRODUCTION Terrestrial 3D laser scanners—a new class of survey instrument—have become popular and are increasingly used in providing as-built and modeling data in transportation applications, including land surveying, archeological studies, architecture, bridge structures, and highway surveys. These scanners measure thousands of data points (distance, angle, and reflected return signal power) per second and generate a very detailed “point cloud” data set. Laser scanner manufacturers suggest this new tool can reduce lane closures, decrease risk of injuries, and increase productivity. The resulting point cloud and detailed 3D model allows engineers to extract all the required data in the office, decreasing or eliminating the need for surveyors to return to the site for additional measurements. Using 3D laser scanners will dramatically improve safety and efficiency over conventional survey methods. However, to fully realize the benefit of using 3D laser scanners, they must be used properly and in appropriate applications. Like any other instrument, the 3D laser scanner has its own set of limitations. The technical specifications of laser scanners as stated by individual manufacturers are typically difficult to reproduce in real-life applications. Generally, laser scanner performance, such as accuracy and detection range, varies with distance, object reflectivity, and angle of incidence to the reflective surface. Currently, each manufacturer provides their scanner specifications differently, using different accuracy terms and often their own trademark terminology. For example, one vendor may specify their target accuracy (one standard deviation) based on a white target at 100 m, while another vendor may specify their TRB 2009 Annual Meeting CD-ROM Paper revised from original submittal. Hiremagalur, Yen, Lasky, and Ravani 2 accuracy (95% confidence level) with no information regarding the target reflectivity or its range. Thus, direct comparison based solely on specifications is nearly impossible. Standards will increase confidence in 3D laser scanner measurements and encourage greater, more consistent, and more effective use of 3D laser scanners. Standard terminology definitions, standard test protocols and metrics, and reporting will enable fair comparisons of instrument capabilities. Despite manufacturers’ and users’ common desire for standardization in terminology and test protocols for 3D terrestrial laser scanners, no standard test protocols for the performance evaluation of 3D laser scanners have been established prior to this research. Lichti et al. (1) were perhaps the earliest to develop tests for calibration of terrestrial laser scanners, including a clear comparison between digital photogrammetry and laser scanning. Balzani et al. (2) followed with accuracy tests in the range direction for terrestrial 3D laser scanners. Boehler et al. (3) installed multiple test targets to investigate the quality of measurements obtained with laser scanners. In addition, Johansson (4) explored the behavior of three different high-resolution ground-based laser scanners in a built environment. Gordon et al. (5) detailed an investigation into the calibration of the Cyrax 2400 3D laser scanner by developing a series of rigorous experiments to quantify the instrument’s precision and accuracy. Fidera et al. (6) used a Cyrax 2500—a pulsed Time-of-Flight (TOF) laser scanner—to study the influence of surface reflectance of different materials on laser scanning, specifically on the maximum coverage angles of cylindrical objects and the resulting determination of their diameter from the point cloud. Jaselskis et al. (7) have performed pilot studies on applying laser scanning for Department of Transportation (DOT) projects for the Iowa DOT using a Cyrax 2500. Kersten et al. (8-10) compared the accuracy of several terrestrial laser scanning systems, and developed accuracy test fixtures and procedures for range accuracy, influence of the laser beam angle of incidence, range noise, influence of color on range measurement, and level compensator accuracy. Sternberg and Kersten (11) examined the workflow in as-builtdocumentation of plants using different scanners, and compared the amount of human labor time for each system (software and hardware) from scanning and registration to complete ComputerAided Design (CAD) model. Cheok et al. (12) provide a status update on the National Institute of Standards and Technology (NIST) work on standards and the National Performance Evaluation Facility for 3D imaging systems. In close cooperation with NIST, the American Society for Testing and Materials (ASTM) International E57 Committee on 3D Imaging Systems has begun a consensus-based standards initiative for 3D imaging systems. DOT applications have unique requirements. Accuracy of the work product carries certain legal implications. Moreover, pavement surveys create extraordinary challenge for laser scanners – measurements are often made at long ranges with large angle of incidences on dark asphalt surfaces. Software must be able to handle “ghost” point cloud images created by passing traffic. Earlier studies lack: • Performance data on latest commercial laser scanners with dual-axis level compensator, • Long-range test data (over 50 m) on and off pavement, • Best practices for laser scanning survey workflow and geo-reference/ registration methodology, and • Point cloud post-processing software evaluation: geo-reference / registration features, Quality Assurance / Quality Control (QA/QC) reporting, and integration to existing CAD software. TRB 2009 Annual Meeting CD-ROM Paper revised from original submittal. Hiremagalur, Yen, Lasky, and Ravani 3 This paper presents a set of vendor-neutral standardized test protocols and metrics for the characterization and performance evaluation of fixed 3D laser scanners for transportation applications. The protocols were designed explicitly to allow users to conduct evaluations in easily accessible facilities. These standardized test protocols and metrics provide a solid scientific and engineering foundation for wide-scale adoption of 3D laser scanners in transportation surveying applications. The paper also clarifies the common limitations of 3D laser scanners, and provides recommended methods for their mitigation. Research Objectives Previous testing and evaluation of 3D laser scanner performance has shown (1, 3, 4, 7, 8, 10, 11) that the performance influencing variables include: laser beam-width, angle of incidence, surface reflectivity and color, range, object edges, and geo-referencing error. Geo-referencing errors are tied to geo-reference methodology, geo-reference target recognition accuracy, and workflow. Moreover, workflow affects worker and public safety, productivity, and the likelihood of human errors in geo-referencing—geo-reference error can far exceed instrument error. Therefore, workflow and geo-referencing methodology is also closely examined here. Our effort by no means replaces NIST’s and ASTM’s standardization efforts. Their work will serve the overall 3D imaging user community. We focused on commercially available TOFbased 3D laser scanners that are relevant to DOT applications. The test goals were to: • Understand 3D laser scanner performance and related influencing variables, • Provide recommended geo-reference / registration methodology and target setup, and • Provide basis for DOT procurement of 3D laser scanners. Our tests were segregated into Control Test and a Pilot Study. The Control Test evaluated 3D laser scanner performance in an outdoor pavement environment with maximum repeatability for the available testing conditions. The Control Test was intended to be repeatable by Caltrans and others, perhaps at a different site, at a later date when new laser scanners become available. The Pilot Study evaluated the use of the 3D laser scanner in a ‘real-world’ Caltrans job scenario at a bridge over State Highway 113 with a clover-leaf ramp on either side (see Figure 1). The Pilot Study, along with the Control Test, clearly illustrated: • The importance of accurate geo-referencing and registration methodologies, • The advantage of reduced targets needed for registration for some scanners that have dual-axis level compensator, • The importance for DOT applications of Field-of-View (FOV) in both the horizontal and vertical plane, and • The importance of high resolution scans for feature identification. This paper presents a subset of the Control Test; for more details, as well as the results of the Pilot Study, see (13), at http://ahmct.ucdavis.edu/images/AHMCT_LidarFinalReport.pdf. TRB 2009 Annual Meeting CD-ROM Paper revised from original submittal. Hiremagalur, Yen, Lasky, and Ravani 4